Shaping Proto-Value Functions Using Rewards

In reinforcement learning (RL), an important sub-problem is learning the value function, which is chiefly influenced by the architecture used to represent value functions. Often, the value function is expressed as a linear combination of a pre-selected set of basis functions. These basis functions are either selected in an ad-hoc manner or are tailored to the RL task using the domain knowledge. Selecting basis functions in an ad-hoc manner does not give a good approximation of value function while choosing functions using domain knowledge introduces dependency on the task. Thus, a desirable scenario is to have a method to choose basis functions that are task independent, but which also provide a good approximation for the value function. In this paper, we propose a novel task-independent method to construct reward-based Proto Value Functions (RPVFs) using the topology of the state space and the reward structure of the underlying RL task. Our methodology uses the connectivity of the state space and the immediate reward structure to construct the basis functions which are required for linear approximation of the value function. The approach we propose gives enhanced learning performance. In particular, when the state space is symmetrical and the value function asymmetrical, the basis functions so constructed capture the asymmetry in value function better than any of the previous approaches. We demonstrate the effectiveness of RPVFs in approximating the value function via experiments on benchmark RL problems as well as on another non-standard problem.